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SUMMARY  PAGE 


PROBLEM: 

To  evaluate  the  computer-assisted  diagnostic  program  for  dental  pain  with  data  collected  by 
Navy  hospital  corspmen,  to  collect  data  on  difficulties  encountered  by  the  corpsmen  in  their  use 
of  the  system,  and  to  obtain  the  corpsmen’s  feedback  regarding  the  usefulness  of  the  system  for 
managing  patients  with  dental  pain. 


FINDINGS: 

The  computer-assisted  dental  program  produced  diagnoses  which  were  exact  or  logically 
consistent  with  the  diagnosis  made  by  the  dentist  83%  of  the  time.  The  corpsmen  would  use  the 
dental  program  in  their  management  of  dental  emergencies  which  occur  at  sea.  They  found  the 
treatment  recommendations  made  by  the  program  especially  helpful.  In  future  revisions,  the 
corpsmen  requested  more  flexibility  in  data  entry,  questions  that  were  more  clearly  worded,  and 
greater  detail  to  definitions  of  terms,  procedures  and  dental  conditions. 


APPLICATION: 

Findings  can  assist  in  the  development  of  a  computer-assisted  dental  diagnostic  system  that 
best  meets  the  needs  of  independent  duty  corpsmen  in  their  management  of  dental  patients  at 
sea. 


ADMINISTRATIVE  INFORMATION 

This  investigation  was  conducted  under  Naval  Medical  Research  and  Development 
Command  Research  Work  Unit  MM33C30.002-5004.  It  was  submitted  for  review  on  May  19, 
1989  and  approved  for  publication  on  June  23,  1989.  It  has  been  designated  as  Naval 
Submarine  Medical  Research  Laboratory  Report  No.  1141. 
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ABSTRACT 


A  computer-assisted  dental  program  to  assist  independent  duty  corpsmen  in  the  diagnosis 
and  management  of  patients  who  present  at  sea  with  dental  pain  produced  diagnoses  which  were 
exact  or  logically  consistent  with  the  diagnosis  made  by  the  dentist  83%  of  the  time.  The 
corpsmen  found  the  computer-assisted  dental  program  valuable  to  the  diagnosis  and  managment 
of  patients  with  dental  pain.  They  would  use  the  the  dental  program  in  their  management  of 
dental  emergencies  which  occur  at  sea.  They  found  the  treatment  recommendations  made  by 
the  program  especially  helpful.  In  future  revisions,  the  corpsmen  requested  more  flexibility  in 
data  entry,  questions  that  were  more  clearly  worded,  and  greater  detail  to  definitions  of  terms, 
procedures,  and  dental  conditions. 


Introduction 


A  computer-assisted  diagnostic  and  treatment  program  for  dental  pain  was  first  developed 
at  the  Naval  Dental  Research  Institute,  Great  Lakes,  IL  1,2  and  later  adapted  to  MS-DOS 
format  and  implemented  on  an  IBM-PC/AT  by  the  Naval  Submarine  Medical  Research 
Laboratory  (NSMRL),  Groton,  CT.  The  program  is  rule-based,  designed  for  use  with  trauma 
and  non-trauma  related  dental  pain  and  also  provides  a  differential  diagnosis  for  soft  tissue 
lesions.  The  program  considers  35  dental  conditions  in  its  evaluation  of  patients  with  trauma 
and  non-trauma  related  dental  pain.  A  separate  module  provides  textbook  style  assistance  to  the 
user  seeking  information  about  soft  tissue  lesions.  The  program  is  intended  for  use  by 
independent  duty  corpsmen  in  their  management  of  patients  who  present  with  dental  pain  at 
sea. 


Symptom  findings  can  be  recorded  on  dental  questionnaires,  or  the  user  can  enter 
information  directly  into  the  computer  program.  Use  of  the  questionnaires  is  optional,  unless 
the  dental  patient  and  computer  program  are  in  separate  locations.  The  questionnaires  permit 
data  to  be  gathered  when  it  is  inconvenient  or  undesirable  to  use  the  computer  at  the  time  of 
the  patient  evaluation. 

Whether  data  are  recorded  on  a  questionnaire  or  entered  directly  into  the  program,  the  user 
must  first  classify  the  patient’s  presenting  problem  into  one  of  ten  categories.  There  is  a 
category  for  trauma  related  dental  pain  and  one  for  the  differential  diagnosis  of  soft  tissue 
lesions.  There  are  eight  categories  of  non-trauma  related  dental  pain.  These  are:  Tooth 
Specific,  Teeth,  Generalized/Multiple;  Gingiva,  Specific  Area;  Gingiva,  Generalized;  Oral 
Mucosa,  Tooth  Associated;  Temporomandibular  Joint/Muscle;  Dental  Extraction  Site;  and  Tissue 
Swelling.  A  different  questionnaire  is  used  for  each  of  the  categories  considered  by  the 
program. 

Each  questionnaire  consists  of  a  set  of  questions  and  branch  points.  The  questions  are  ones 
used  by  the  program  to  collect  patient  data.  The  branch  points  in  the  questionnaire  reflect  the 
implicit  logic  of  the  diagnostic  program  and  direct  the  user  to  gather  further  information 
depending  on  the  patient’s  responses  to  previous  questions.  If  the  corpsman  follows  the 
directions  of  the  branch  points  he  will  respond  only  to  those  questions  required  by  the  computer 
program  to  reach  a  diagnosis. 

The  user  enters  the  patient’s  symptom  findings  into  the  program.  To  make  a  diagnosis,  the 
program  compares  dental  findings  against  a  set  of  diagnostic  rules  and  makes  a  diagnosis  when, 
and  if,  the  conditions  of  a  rule  are  met.  Sometimes  the  program  cannot  reach  a  diagnosis  based 
on  the  information  provided,  and,  in  these  cases,  a  statement  to  that  effect  is  displayed. 

Diagnostic  rules  for  the  dental  program  were  compiled  from  dentists  and  the  dental 
literature-1  The  system  was  initially  reviewed  by  seven  dentists  who  generated  over  200 
simulated  dental  emergencies.  Simulated  cases  were  evaluated  by  the  computer-assisted 
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diagnostic  program,  and,  based  upon  the  results,  minor  changes  were  made  to  the  rules. 
Subsequently,  the"  dental  program  was  tested  in  a  large  Navy  dental  clinic.  Data  were  collected 
by  a  dental  technician,  and  a  diagnostic  accuracy  of  88%  was  obtained  for  80  real  patients  on  a 
subset  of  dental  conditions  considered  by  the  program. 

The  dental  diagnostic  program  was  further  evaluated  in  two  studies  conducted  at  NSMRL.^ 
In  these  studies,  dentists  supplied  "classic"  findings  for  each  of  the  35  dental  conditions 
considered  by  the  program.  When  the  "classic"  findings  for  each  condition  were  entered  into  the 
diagnostic  program,  the  program  produced  diagnoses  which  agreed  with  the  "classic"  diagnosis 
78%  of  the  time.  Accuracy  was  also  evaluated  according  to  the  therapeutic  interventions 
available  to  the  corpsman  at  sea.  These  are:  (a)  evacuate  immediately;  (b)  treat  aboard  - 
potential  for  condition  to  warrant  evacuation;  and  (c)  treat  aboard  -  return  to  duty.  The 
program  correctly  distinguished  between  dental  conditions  requiring  immediate  evacuation  and 
those  which  can  reasonably  be  managed  aboard  the  submarine  by  the  corpsman. 

Up  to  now  the  program  has  not  been  used  by  Navy  corpsmen.  The  purpose  of  the  present 
study  is  to  collect  data  on  difficulties  encountered  by  corpsmen  in  using  either  the  dental 
questionnaire  or  diagnostic  program,  to  obtain  the  corpsmen’s  feedback  regarding  the  usefulness 
of  the  system  for  managing  patients  with  dental  pain,  and  to  gather  the  corpsmen’s  suggestions 
for  improving  either  the  dental  questionnaires  or  diagnostic  program.  This  information  will  be 
used  to  create  a  program  that  best  meets  the  needs  of  corpsmen  in  their  management  of  dental 
patients  at  sea. 


Subjects 

Five  Navy  corpsmen  participated  in  the  study.  One  was  a  submarine  qualified,  independent 
duty  technician;  two  were  field  medical  technicians,  one  was  a  cardio-pulmonary  technician  and 
one  was  an  optical  technician.  Two  were  chief  petty  officers,  one  was  a  1st  class  petty  officer, 
and  one  was  a  2nd  class  petty  officer.  Since  the  system  had  never  been  tested  by  any  group  of 
corpsmen,  we  felt  that  the  opportunity  for  corpsmen  to  comment  and  report  on  difficulties 
encountered  in  using  the  system  outweighed  any  disadvantage  due  to  specific  differences  in 
training. 


Procedure 

Over  a  two  day  period,  the  corpsmen  were  trained  both  in  the  use  of  the  dental 
questionnaires  for  collecting  patient  information  and  in  the  use  of  the  diagnostic  program  for 
obtaining  a  diagnosis.  Training  focused  on  the  questions  used  to  collect  patient  information, 
dental  terminology,  and  diseases  considered  by  the  program.  In  addition  to  classroom 
instruction,  the  corpsmen  received  approximately  four  hours  of  hands-on  training  in  the 
examination  procedures  required  by  the  dental  questionnaires  by  the  dental  officer  responsible 
for  the  dental  curriculum  at  the  Naval  Undersea  Medical  Institute,  Groton,  CT.  He  stated  that 
the  training  provided  the  corpsmen  with  dental  skills  and  knowledge  equivalent  to  the  corpsmen 
trained  for  independent  duty  aboard  submarines. 
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After  completion  of.  training,  each  corpsman  was  assigned  one  morning  a  week  for  a  month 
to  the  Branch  Dental  Clinic,  NAVSUBASE  NLON,  Groton,  CT.  When  a  patient  in  acute  dental 
pain  presented  at  the  clinic,  he  was  triaged  by  a  dentist  to  determine  if  he  required  immediate 
attention  or  if  he  could  tolerate  first  being  evaluated  by  the  corpman.  This  decision  was 
determined  primarily  by  the  patient’s  level  of  pain;  if  the  patient  was  holding  his  head,  crying, 
or  in  obvious  severe  pain  he  was  given  immediate  attention.  The  triage  interview  was  as  brief 
as  possible  to  minimize  bias  in  the  patient’s  responses  and  the  corpsman  was  not  present  during 
the  triage.  The  dentist  asked  the  patient  a  few,  non-specific  questions  regarding  the  location, 
possible  cause  and  duration  of  pain.  The  dentist  did  not  respond  to  patient  questions  or  offer  a 
tenative  diagnosis.  Dental  patients  were  male  and  female,  active  duty  and  retired  personnel,  17 
years  or  older  in  acute  dental  pain. 

Corpsmen  used  a  dental  questionnaire  as  a  guide  for  interviewing  the  patient.  Dental 
questionnaires  consist  of  a  list  of  questions  and  branch  points.  The  branch  points  direct  the 
corpsman  to  gather  further  information  depending  on  the  patient’s  responses  to  previous 
questions.  Branch  points  eliminate  the  need  for  the  corpsman  to  complete  all  items  on  a  given 
questionnaire.  By  following  the  directions  provided  by  the  branch  points,  the  corpsman  will 
collect  only  that  information  required  by  the  computer  program  for  a  diagnosis.  The  corpsman 
assessed  the  nature  of  the  dental  problem  on  the  basis  of  the  patient’s  complaint  clarified  by  a 
standard  history  and  a  cursory  exam.  The  corpsman  then  selected  the  appropriate  questionnaire 
to  begin  a  structured  history  and  physical  exam  as  directed  by  the  questions  and  branch  points. 
The  questionnaire  sometimes  required  the  corpsman  to  obtain  information  from  the  patient’s 
dental  record  and/or  examine  the  patient’s  mouth  with  the  aid  of  an  intra-oral  mirror.  In 
addition,  the  corpsman  was  sometimes  required  to  perform  routine,  non-invasive  examinations, 
i.e.,  percussion  and  palpation  of  the  affected  area.  Normally,  the  examination  was  completed  in 
ten  to  fifteen  minutes. 

After  completing  the  dental  questionnaire,  the  corpsman  made  his  own  diagnosis  without 
the  aid  of  the  computer.  He  was  encouraged  to  select  from  among  the  35  diagnoses  considered 
by  the  program  but,  if  he  felt  another  was  more  appropriate,  he  specified  the  diagnosis.  The 
corpsman  directed  the  patient  to  the  dentist  and  then  entered  the  patient’s  responses  to  the 
questionnaire  into  the  computer-assisted  dental  program.  The  corpsman  was  asked  to  note  any 
difficulties  he  encountered  in  either  administering  the  questionnaire  or  using  the  computer 
program. 

The  dentist  examined  and  treated  the  patient.  Based  on  his  examination,  the  dentist 
selected  a  diagnosis(es)  from  the  list  of  35  diagnoses  considered  by  the  dental  program.  If  he 
felt  the  diagnosis  was  other  than  one  considered  by  the  program,  he  specified  the  diagnosis. 

The  dentist  also  classified  the  patient’s  dental  problem  into  one  of  ten  broad  categories  used  by 
the  program  and  which  correspond  to  the  ten  questionnaires  available  to  the  corpsman. 

At  the  end  of  the  study  each  corpsman  completed  a  six  page  form  evaluating  the  dental 
questionnaire  and  diagnostic  program.  The  corpsmen  rated  characteristics  of  the  questionnaire 
and  diagnostic  program  on  a  six  point  scale  from  strongly  agree  to  strongly  disagree.  Items 
concerned  clarity  of  questions,  familiarity  with  dental  terminology  and  examination  procedures 
required  by  the  system,  perceived  usefulness  of  the  system  for  managing  dental  patients,  and 
trust  in  the  diagnostic  and  treatment  suggestions  made  by  the  program.  A  copy  of  this  form  is 
included  as  Appendix  A. 
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Results 


Dental  Conditions  Examined 

In  this  study  the  corpsmen  examined  32  patients  with  acute  dental  pain.  There  was  no 
instance  when  a  patient  was  excluded  from  the  study  because  the  delay  in  using  the 
questionnaire  would  have  unacceptably  delayed  treatment.  All  the  corpsmen’s  examinations 
were  conducted  prior  to  an  examination  by  the  dentist.  The  range  of  dental  problems  which 
were  brought  to  the  clinic  were  the  common  causes  of  acute  dental  pain.  Table  1  lists  the 
dental  conditions  considered  by  the  program  and  the  number  of  times  each  condition  was  seen 
during  the  study. 


Table  1.  Thirty- five  dental  conditions  considered  by  the  trauma  and  non-trauma  related  dental 
program  and  the  number  of  times  each  dental  condition  was  diagnosed  by  a  dentist.* 


1  Abscess/infection/cellulitis 

_ Acute  apical  abscess 

1  Acute  apical  periodontitis 

_ Acute  herpetic  gingivostomatitis 

1  Acute  gingivitis 
JLCarious  lesion  (decay) 

1  Dentin  hypersensitivity 

1  Defective  restoration 

_ Displace/mobility  of  tooth,  favorable  prognosis 

_ Displace/mobility  of  tooth,  guarded  prognosis 

_2_Endo/perio  combined  problem 

2  Enamel  fracture 
_ Food  impaction 

_ Fractured  crown,  small  pulp  exposure 

_ Fractured  crown,  large  pulp  exposure 

1  Fractured  crown,  pulp  not  exposed 
_ Fractured  alveolar  bone 


_ Fractured  mandible 

_ Fractured  maxilla 

_ Fractured  facial  bones 

6  Irreversible  pulpitis 

_ Internal  derangement  of  the  TMJ 

_ Localized  alveolar  osteitis 

_ Maxillary  sinusitis 

_ Myofascial  pain/muscle  spasms 

_2  Necrotizing  ulcerative  gingivitis 

_ Neurologic  injury 

_ Osseous  sequestrum 

2  Occlusal  trauma 
_L  Periodontal  abscess 

3  Periocoronitis/eruoting  tooth 
2  Reversible  pulpitis 

_ Root  fracture 

_ Total  avulsion,  good  candidate  for  replant 

Total  avulsion,  poor  candidate  for  replant 


* 

There  are  only  29  diagnoses  because  on  3  of  the  32  cases  the  dentist  made  a  diagnosis  of 
apthous  ulcer,  a  condition  included  in  the  module  for  the  differential  diagnosis  of  soft  tissue 
lesions.  This  module  provides  textbook  style  assistance  about  soft  tissue  lesions. 
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Corosman  -  Dentist  Agreement 

The  presenting  dental  problem  of  each  patient  was  classified  by  the  examining  dentist  into 
one  of  ten  categories  corresponding  to  the  ten  sections  of  the  diagnostic  program.  Table  2  lists 
these  categories.  Next  to  each  category  are:  a)  the  number  of  patients  classified  by  the  dentist 
into  that  category,  b)  the  number  of  patients  classified  by  the  corpsmen  into  that  category,  and 
c)  the  number  of  times  the  corpsmen’s  selection  corresponded  to  the  category  selected  by  the 
dentist.  For  example,  of  the  32  patients  examined,  the  dentist  identified  18  patients  with  a 
problem  related  to  a  specific  tooth.  On  17  of  these  patients,  the  corpsmen  appropriately 
selected  the  "Tooth  Specific"  dental  questionnaire  to  use  in  interviewing  the  patient.  For  the 
remaining  patient,  the  corpsmen  selected  a  different  questionnaire.  The  corpsmen  used  the 
tooth  specific  questionnaire  for  26  of  the  32  patients  in  this  study. 


Table  2.  Dental  categories,  the  number  of  patients  classified  by  the  dentist  and  corpsmen  (HM) 
into  each  category,  and  the  number  of  times  the  corpsmen’s  questionnaire  selection  corresponded 
to  the  one  made  by  the  dentist. 


Dental  Category 


Soft  Tissue  Lesion 
Trauma  Related  Injury 
♦Tooth,  Specific 
♦Teeth,  Generalized 
♦Gingiva,  Specific  Area 
♦Gingiva,  Generalized  Area 
♦Oral  Mucosa,  Tooth  Associated 
♦Temporomandibular  Joint/Muscle 
♦Dental  Extraction  Site 
♦Tissue  Swelling 

*  Non-trauma  related  category. 


Dentist 

HM 

HM-Dentist 

Correspondence 

3 

1 

1 

1 

1 

0 

18 

26 

17 

1 

1 

0 

6 

1 

1 

2 

2 

2 

1 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

Branching  Errors 

Errors  in  following  the  directions  provided  by  the  branch  points  were  identified  by 
examining  the  completed  questionnaires.  For  12  of  the  32  dental  patients,  the  corspman  made 
no  errors  in  interpreting  the  directions  given  by  the  branch  points.  On  the  remaining  20  cases, 
the  corpsmen  made  1  error  (on  12  cases),  2  errors  (on  3  cases),  or  3  errors  (on  5  cases).  In  most 
cases,  the  errors  resulted  in  the  collection  of  more  information  than  was  required  by  the 
computer  to  make  a  diagnosis.  However,  on  three  cases,  the  errors  resulted  in  the  failure  to 
collect  the  required  information  and,  as  a  result,  the  computer  was  unable  to  make  a  diagnosis 
for  these  cases. 
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Computer  Diagnoses 


Patient  data  collected  by  the  corpsmen  were  entered  into  the  computer  and  the  diagnoses 
generated  by  the  computer  were  recorded.  The  computer  makes  a  diagnosis  when,  and  if,  the 
symptom  findings  match  the  requirements  of  a  diagnostic  rule.  It  is  not  only  possible  but 
common  for  the  program  to  arrive  at  multiple  diagnoses.  In  general,  the  other  diagnoses  which 
accompany  the  "correct"  diagnosis  are  often  related,  if  not  causal,  to  the  patient’s  dental 
condition.  In  this  study,  the  computer  program  arrived  at  a  single  diagnosis  on  eight  cases, 
arrived  at  two  diagnoses  on  six  cases,  and  made  three  or  more  diagnoses  on  the  remaining  cases. 

The  diagnoses  made  by  the  dentist  were  accepted  as  the  final  diagnoses  against  which  the 
computer-generated  diagnoses  and  the  diagnoses  made  by  the  corpsman  were  compared.  The 
degree  of  agreement  between  the  computer,  the  corpsman  and  the  dentist  was  defined  according 
to  the  classification  schema  used  in  an  earlier  evaluation  of  the  system  .  This  schema  classifies 
agreement  into  three  categories:  1)  exact  match  or  logically  consistent  diagnosis;  2)  non-match; 
and  3)  other.  A  logically  consistent  diagnosis,  though  not  exact  match,  is  defined  as  a  diagnosis 
consistent  with  the  information  entered  into  the  program  and  for  which  the  recommended 
treatment  is  consistent  with  the  emergency.  Diagnoses  were  classified  as  "other"  when  the  data 
collected  by  the  corpsman  was  contradictory  or  illogical  either  because  of  patient  or  corpsman 
error. 

The  computer  made  a  diagnosis  on  29  of  the  32  cases.  On  the  remaining  3  cases,  the 
corpsmen  failed  to  collect  the  information  required  by  the  computer  for  a  diagnosis.  The 
computer-generated  diagnosis  was  exact  or  logically  consistent  with  the  dentist’s  diagnosis  on  24 
cases  (83%).  On  the  remaining  5  cases,  2  were  non-matches  (7%),  and  3  were  classified  as  other 
(10%).  The  corpsman  recorded  a  diagnosis  on  32  cases.  The  corpsman’s  diagnosis  unaided  by  a 
computer  was  exact  or  logically  consistent  with  the  dentist’s  diagnosis  on  22  cases  (69%).  Ten 
of  the  corpsmen’s  diagnoses  were  non-matches  (31%).  None  was  classified  as  other. 


Corosmen’s  Evaluation  of  the  Questionnaire  and  Program 

At  the  completion  of  the  study,  each  corpsman  completed  a  form  evaluating  the  dental 
questionnaires  and  diagnostic  program.  The  corpsmen  read  statements  regarding  the  computer- 
assisted  dental  system  and  rated  their  agreement  with  the  statement  on  a  six  point  scale.  The 
items  addressed  clarity  of  questions,  familiarity  with  dental  terminology,  perceived  usefulness  of 
the  system,  and  trust  in  diagnostic  and  treatment  suggestions.  A  tally  of  the  corpsmen’s  ratings 
for  each  item  is  given  in  Table  3.  The  corpsmen  also  identified  problems  they  encountered  and 
suggestions  for  improving  the  diagnostic  system.  The  corpsmen’s  comments  are  in  italics  in 
Appendix  A. 
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Table  3.  The  number  of  corpsmen  (n=5)  who  marked  each  rating  on  items  evaluating  the  dental 
questionnaires,  branch  points  and  the  computer-assisted  dental  program.  The  bold-faced  letter  code 
preceding  each  item  identifies  whether  the  item  addressed  clarity  of  questions  (C);  familiarity  with 
dental  terms /procedures^);  perceived  usefulness  of  the  system  (PU);  or  trust  in  diagnostic  and 


treatment  recommendations  (T). 

Strongly 

Agree 

1 

2 

3 

4 

5 

Strongly 

Disagree 

6 

Dental  Questionnaires 

(C)  Questions  clearly  worded 

V 

2 

2 

(C)  Questions  in  logical  order 

2 

2 

1 

- 

- 

- 

(F)  Familiar  with  terms  in  questionnaires 

1 

4 

- 

- 

- 

- 

(PU)  Questionnaires  useful  for  evaluating  dental  pain 

2 

- 

1 

1 

1 

- 

(F)  More  training  required  to  perform  procedures 

required  by  questionnaires 

1 

1 

3 

1 

2 

_ 

(PU)  Hard  to  select  the  most  appropriate  questionnaire 

1 

- 

1 

- 

2 

1 

(PU)  Questionnaires  are  superfluous 

2 

2 

— 

1 

— 

— 

Branch  Points 

(C)  Branch  points  clearly  worded 

1 

1 

1 

2 

(C)  Branch  directions  easy  to  follow 

3 

1 

1 

- 

- 

- 

(PU)  Eliminate  branch  points 

- 

1 

— 

— 

3 

1 

Comouter-assisted  Dental  Program 

(PU/T)  Use  program  for  routine  evaluations* 

lb 

2b,c 

ld 

le 

(PU/T)  Use  program  only  for  emergencies* 

ld 

- 

la 

le 

- 

(F)  HM  are  sufficiently  trained  to  use  program 

1 

2 

- 

- 

1 

1 

(F)  Unfamiliar  with  terms  used  by  program 

- 

1 

1 

1 

2 

- 

(PU/T)  Tx  recommendations  not  suitable  at  sea 

- 

- 

2 

1 

1 

1 

(PU)  Classification  of  diagnoses  into  possible 

and  probable  helpful 

2 

2 

1 

_ 

_ 

_ 

(T)  I  trust  the  program’s  diagnoses 

- 

2 

1 

1 

1 

- 

(T)  CO/XO  would  trust  the  program’s  diagnoses 

1 

1 

2 

- 

1 

- 

(PU/T)  I  would  consider  tx  recommendations  in 
managing  dental  patients  at  sea 

3 

2 

_ 

_ 

- 

(F)  Program  provides  adequate  definitions 

2 

- 

- 

2 

1 

- 

(PU)  Program  is  a  good  supplement  to  dental 

textbooks**  _ 

2 

1 

_ 

- 

_ 

(PU)  At  sea,  dental  text  should  be  in  computer 

not  book  format 

2 

2 

1 

*The  responses  made  by  each  corpsmen  are  identified  by  the  letters  a-e.  Correct  interpretation  requires 
knowledge  of  the  individual’s  combined  ratings  for  these  two  questions. 

** 

Two  ratings  were  missing  for  this  item. 
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Clarity  of  Questions 

Four  items  concerned  the  clarity  of  questions  or  branch  points  used  by  the  dental 
questionnaires  and  diagnostic  program.  These  items  are  marked  as  "C"  (clarity)  on  the  form 
included  in  Appendix  A. 

Generally,  the  corpsmen  felt  that  the  directions  given  in  the  branch  points  were  clearly 
worded  and  easy  to  follow.  While  the  corpsmen  felt  that  the  questions  used  to  collect 
information  from  the  patient  followed  a  logical  order,  some  of  the  corpsmen  felt  that  they  could 
be  more  clearly  worded. 

Familiarity  with  Terms  and  Procedures 

Five  items  addressed  the  corpsmen’s  familiarity  with  the  terminology  and  the  examination 
procedures  required  by  the  dental  system.  These  items  are  marked  as  "F"  (Familiarity)  on  the 
form  (Appendix  A). 

The  corpsmen  rated  themselves  as  familiar  with  the  terminology  used  by  the  program.  In 
their  comments,  they  noted  that  the  training  of  the  independent  duty  technician  is  sufficient  to 
perform  the  examinations  procedures  required  by  the  program.  Some  of  the  corpsmen  felt  that 
the  definition  of  terms  or  dental  conditions  provided  by  the  program  was  inadequate  and  could 
be  improved  by  more  detail. 


Perceived  Usefulness  Of  System 

Eleven  items  addressed  the  perceived  usefulness  of  the  dental  questionnaires  and  the 
diagnostic  system  to  the  management  of  patients  who  present  at  sea  with  dental  problems. 

These  items  are  marked  as  "PU"  (perceived  usefulness)  on  the  form  (Appendix  A). 

These  items  showed  that  the  corpsmen  found  the  questionnaire  superfluous  and  would  prefer 
to  enter  patient  information  directly  into  the  computer.  In  addition,  some  of  the  corpsmen  had 
difficulty  deciding  which  questionnaire  to  use  to  conduct  the  interview.  Rather  than  answering 
all  the  items  on  the  questionnaire,  the  corpsmen  preferred  to  follow  the  directions  in  the  branch 
points  and  answer  only  the  items  required  by  the  computer  for  a  diagnosis. 

Items  which  addressed  the  perceived  usefulness  of  the  diagnostic/treatment  program  showed 
that  all  but  one  corpsman  would  use  the  system  for  the  diagnosis  and  management  of  serious 
dental  emergencies  and  that  some  of  these  corpsmen  would  also  use  the  system  in  their  routine 
evaluation  of  patients  at  sea  with  dental  pain.  One  corpsman  did  not  find  the  system  useful. 

He  felt  that  the  books  available  to  the  corpsmen  aboard  a  submarine  provide  sufficient 
information.  All  of  the  corpsmen  would  consider  the  treatment  recommendations  made  by  the 
program  in  their  management  of  patients.  While  the  corpsmen  generally  felt  that  the  program 
provided  a  good  supplement  to  the  textbooks  available  aboard  the  submarine,  opinion  varied 
greatly  on  whether  or  not  to  have  all  dental  text  in  computer  rather  than  book  format. 
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Trust  in  Diagnostic  and  Treatment  Recommendations 

Six  items  addressed  the  corpsmen’s  trust  in  the  diagnostic  and/or  treatment 
recommendations  made  by  the  system  for  the  management  of  dental  patients.  These  items  are 
marked  "T"  (Trust)  on  the  form  (Appendix  A). 

The  corpsmen’s  trust  in  the  dental  diagnostic  system  was  variable.  None  of  the  corpsmen 
either  strongly  trusted  or  distrusted  the  diagnoses  made  by  the  system;  they  were  more  on  the 
neutral  side  in  their  evaluation.  It  is  interesting  that  all  but  one  corpsman  felt  that  the 
Commanding  and  Executive  Officers  of  the  submarine  would  trust  the  diagnoses  made  by  the 
program.  It  is  also  interesting  that  while  the  corpsmen  were  not  certain  of  their  trust  in  the 
system,  all  would  take  into  consideration  the  treatment  recommendations  made  by  it. 


Discussion 


The  computer  generated  a  diagnosis  which  was  exact  or  logically  consistent  with  the 
diagnosis  made  by  the  dentist  83%  of  the  time.  The  findings  reported  here  are  in  agreement 
with  previous  studies  reporting  a  diagnostic  accuracy  of  88%  for  80  real  patients  and  78%  for  79 
simulated  cases.  While  earlier  studies  used  data  collected  by  either  Navy  dental  technicians  or 
dentists,  this  is  the  first  time  that  the  accuracy  of  the  dental  program  has  been  evaluated  with 
data  collected  by  Navy  hospital  corpsmen. 

The  primary  purpose  of  this  study  was  to  learn  of  difficulties  encountered  by  corpsmen  in 
using  the  dental  program  and  to  obtain  their  suggestions  for  improving  the  system  to  best  meet 
the  needs  of  the  independent  duty  technicians  at  sea.  The  corpsmen  had  difficulty  both  in 
selecting  and  using  the  dental  questionnaires  to  guide  their  interview.  Except  for  tooth  specific 
problems,  the  corpsmen’s  selection  of  a  dental  questionnaire  rarely  corresponded  to  the  dental 
category  selected  by  the  dentist.  In  addition,  the  corpsmen  made  errors  in  following  the 
directions  given  by  the  branch  points.  In  a  few  cases,  these  errors  resulted  in  the  collection  of 
insufficient  information  for  the  computer  to  reach  a  diagnosis.  In  their  subjective  ratings  of  the 
system,  the  corpsmen  did  not  like  using  the  dental  questionnaires  and  preferred  to  enter 
information  directly  into  the  computer  program.  Direct  entry  avoids  the  problem  of  collecting 
unnecessary  or  insufficient  information  as  the  program  guides  the  user  from  question  to 
question. 

Whether  the  corpsman  records  symptom  findings  on  a  questionnaire  or  enters  information 
directly  into  the  program,  he  must  first  classify  the  patient’s  dental  problem  into  one  of  ten 
categories.  Accurate  diagnosis  by  the  program  is  dependent  on  user  classification  of  the  patient 
into  the  appropriate  dental  category.  In  this  study,  the  corpsmen  usually  classified  the  patient’s 
presenting  dentab  problem  as  one  related  to  a  specific  tooth.  The  dentist  and  corpsmen  failed  to 
agree  on  their  classification  of  the  dental  problem  for  one-third  of  the  dental  patients.  The 
corpsmen  would  clearly  benefit  from  additional  assistance  in  the  classification  of  the  presenting 
dental  problem.  For  example,  the  program  could  be  modified  to  use  data  (e.g.  location, 
duration  and  type  of  pain)  collected  in  an  initial  evaluation  of  the  patient  to  classify  the  patient 
into  the  most  appropriate  category. 
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Despite  their  difficulties,  the  corpsmen  liked  the  program.  Most  of  them  would  use  the 
dental  program"!!!  the  diagnosis  and  management  of  serious  dental  emergencies,  but  not  always 
in  their  evaluation  of  routine  dental  problems.  The  corpsmen  felt  that  when  they  were  . 
confident  of  their  own  diagnosis  and  treatment  plan,  use  of  the  program  would  be  a  waste  of 
time.  When  they  were  less  sure,  they  would  welcome  the  assistance.  While  the  corpsmen  did 
not  always  trust  the  diagnoses  made  by  the  program,  they  would  always  take  into  consideration 
the  treatment  recommendations  made  by  the  dental  program.  The  corpsmen  valued  the  dental 
diagnostic  system  as  a  "second  opinion”.  One  corpsman  felt  that  the  program  kept  him  thinking 
"in  the  right  direction".  Another  corpsman  thought  the  system  might  give  the  Commanding  and 
Executive  Officers  more  confidence  in  the  corpsman’s  management  of  the  case.  The  corpsmen 
found  the  treatment  and  management  recommendations  provided  by  the  program  especially 
helpful. 

The  corpsmen  were  asked  to  consider  the  disadvantages  of  the  program.  One  concern  was 
that  a  corpsman  who  is  unsure  of  his  own  skill  may  rely  too  heavily  on  the  program,  rather  than 
using  it  as  a  diagnostic  aid.  Two  of  the  corpsmen  felt  that  the  evaluation  of  most  dental  pain 
does  not  require  the  assistance  of  a  computer  diagnostic  program.  They  questioned  whether  the 
need  justified  the  expense  of  the  system. 

The  corpsmen  suggested  ways  to  improve  the  program.  The  current  version  of  the  dental 
program  does  not  allow  the  user  to  return  to  previous  pages,  either  to  review  or  to  make 
changes  in  data  entry.  Instead,  the  user  must  return  to  an  earlier  menu  and  begin  data  entry  for 
that  patient  from  the  start.  The  corpsmen  suggested  that  the  user  be  allowed  to  proceed  to  and 
from  successive  pages,  providing  more  flexibity  in  data  entry.  Symptom  information  is 
collected  about  the  patient  by  responding  to  a  set  of  questions.  The  corpsmen  found  the 
wording  of  some  of  the  questions  unclear  and  suggested  rewriting  them.  Definitions  of  terms, 
procedures,  and  dental  conditions  can  be  accessed  by  the  user  from  within  the  computer 
program.  The  corpsmen  found  the  definitions  inadequate  and  suggested  that  they  be  written  in 
greater  detail.  These  suggestions  are  included  in  projected  enhancements  of  this  program  for 
future  release  from  this  laboratory. 


Summary 


Navy  hospital  corpsmen  conducted  interviews  of  32  patients  with  acute  dental  pain. 
Symptom  findings  were  entered  into  a  computer-assisted  program  for  the  diagnosis  and 
management  of  dental  conditions.  The  computer  generated  a  diagnosis  which  was  exact  or 
logically  consistent  with  the  diagnosis  made  by  the  dentist  83%  of  the  time. 

While  only  five  corpsmen  participated  in  this  study,  it  is  our  belief  that  the  views  of  these 
corpsmen  are  representative  of  Navy  hospital  corspmen  as  a  group.  The  corpsmen  found  the 
computer-assisted  dental  program  valuable  to  the  diagnosis  and  management  of  patients  with 
dental  pain.  They  would  use  the  program  in  their  management  of  dental  emergencies  which 
occur  at  sea  and  found  the  treatment  recommendations  especially  helpful.  In  future  revisions, 
the  corpsmen  requested  more  flexibility  in  data  entry,  questions  that  were  more  clearly  worded, 
and  greater  detail  to  definitions  of  terms,  procedures  and  dental  conditions. 
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APPENDIX  A 


The  following  form  was  used  to  evaluate  the  dental  questionnaire  and  diagnostic  program. 
The  items  addressed  clarity  of  questions,  familiarity  with  dental  terminology,  perceived 
usefulness  of  system,  and  trust  in  diagnostic  and  treatment  suggestions.  Next  to  each  item  is  a 
letter  code  identifying  the  type  of  item.  They  are  C:  Clarity  of  Questions;  PU:  Perceived 
Usefulness  of  System;  F:  Familiarity  with  Terms  and  Procedures;  and  T:  Trust  in  Diagnostic 
and  Treatment  Recommendations.  The  letter  codes  did  not  appear  on  the  form  completed  by 
the  corpsmen.  The  corpsmen’s  comments  and  suggestions  in  Sections  II  and  IV  are  in  italics. 


SECTION  I 

The  following  questions  relate  to  the  dental  questionnaires.  We  are  interested  in  evaluating  the 
usefulness  of  this  form  for  conducting  an  interview  of  patients  with  dental  pain.  Your 
responses  will  be  help  to  create  a  form  that  can  be  used  by  Independent  Duty  Corpsmen  at  sea 
in  their  management  of  dental  injuries.  Please  answer  each  item  by  circling  the  number  which 
best  corresponds  to  your  response. 


Example: 

I  like  my  car. 

1  2  3  4  5  6 

Strongly  Agree  Strongly  Disagree 

Since  I  drive  a  beaten  up  old  car  and  I  don’t  like  old  cars,  I  circle  the  number  6  indicating  that 
I  strongly  disagree  with  the  statement  "I  like  my  car”. 


C:  1.  Questions 

were  clearly  worded. 

1 

Strongly  Agree 

2  3  4 

5 

6 

Strongly  Disagree 

C:  2.  Questions 

followed  a  logical  order. 

1 

Strongly  Agree 

2  3  4 

5 

6 

Strongly  Disagree 

F:  3.  Corpsmen  are  familiar  with  most  of  the  terminology  used  in  the  dental  questionnaires. 

1 

Strongly  Agree 

2  3  4 

5 

6 

Strongly  Disagree 

PU:  4.  The  dental  questionnaires  would  be  useful  to  corpsmen  at  sea  in  evaluating  patients 
with  dental  pain. 

1  2  3  4  5  6 

Strongly  Agree  Strongly  Disagree 
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F:  5.  Corpsmen  need  supplementary  training  to  perform  examination  procedures  required  by 
the  dental  questionnaires. 

1  2  3  4  5  6 

Strongly  Agree  Strongly  Disagree 


PU:  6.  It  was  difficult  to  decide  which  of  the  ten  questionnaires  to  use  in  interviewing  the 
dental  patient. 

1  2  3  4  5  6 

Strongly  Agree  Strongly  Disagree 


PU:  7.  Dental  questionnaires  are  superfluous.  I  would  prefer  to  enter  data  directly  into  the 
computer  based  dental  program. 

1  2  3  4  5  6 

Strongly  Agree  Strongly  Disagree 


In  the  dental  questionnaires,  branch  points  were  printed  in  bold  face  and  direct  the  user  to 
subsequent  questions  based  on  response(s)  to  previous  questions.  While  most  branch  points  are 
straight  forward  because  the  branch  is  based  on  a  response  to  a  single  question,  some  are  more 
complicated.  Branch  points  allow  the  user  to  skip  some  of  the  questions.  An  example  of  a 
branch  point  is:  "on  QUESTION  3,  IF  "NO"(2),  THEN  GO  TO  QUESTION  5".  The  following  set  of 
questions  concern  branch  points. 

C:  1.  Branch  points  were  clearly  worded. 

1  2  3  4  5  6 

Strongly  Agree  Strongly  Disagree 


C:  2.  The  directions  given  by  the  branch  points  were  easy  to  follow. 

1  2  3  4  5  6 

Strongly  Agree  Strongly  Disagree 


PU:  3.  It  would"  be  better  to  eliminate  branch  points,  although  it  would  mean  answering  all 
items  on  the  questionnaire. 

1  2  3  4  5  6 

Strongly  Agree  Strongly  Disagree 
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SECTION  II 


We  want  to  he&F-your  opinion  of  the  dental  questionnaires.  Your  comments  and  suggestions 
will  be  used  to  improve  the  dental  questionnaires. 


1.  What  problems  do  you  foresee  in  administering  the  dental  questionnaires.  Please,  be 
specific. 

Eliminate  soft  tissue  section ,  because  the  corpsman  can  not  make  a  determination  of  the 
diagnosis  as  would  a  doctor  or  dentist. 

I  had  no  problems.  Questionnaires  are  an  unnecessary  step  when  the  computer  asks  the 
same  questions. 

Following  the  directions  in  the  branch  points  was  annoying.  It  is  better  to  go  straight 
through  the  program. 

The  questionnaires  would  be  superfluous  if  the  computer  is  available.  Once  I  became 
more  experienced  with  the  branch  points,  I  found  the  questionnaire  easier  to  use. 


2.  Do  you  have  any  suggestions  to  improve  the  dental  questionnaires?  Be  specific. 


Make  some  of  the  questions  more  specific. 

Get  rid  of  the  questionnaires  and  work  directly  with  the  computer. 


3.  Other  comments. 

The  training  given  independent  duty  technicians  covers  dental  examination  procedures. 
Independent  duty  technicians  are  trained  in  the  procedures  required  by  the  dental  program. 
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SECTION  III 


The  following  questions  concern  the  computer  based  diagnostic  program  for  dental  injuries.  We 
are  interested  in  hearing  your  feelings  regarding  the  usefulness  of  this  diagnostic  program  to 
Navy  Hospital  Corpsmen.  Your  responses  will  help  us  to  create  a  program  that  can  be  used  by 
Independent  Duty  Corpsmen  at  sea  in  their  management  of  dental  emergencies.  Please  answer 
each  item  by  circling  the  number  which  best  corresponds  to  your  response. 


PU/T:  1.  I  would  use  this  program  in  my  routine  evaluation  of  patients  at  sea  with  dental 
problems.* 

1  2  3  4  5  6 

Strongly  Agree  Strongly  Disagree 

PU/T:  2.  I  would  use  this  program  only  for  the  diagnosis  and  managment  of  serious  dental 
emergencies.* 

1  2  3  4  5  6 

Strongly  Agree  Strongly  Disagree 


F:  3.  Navy  Corpsman  are  trained  in  the  examination  procedures  required  by  the  diagnostic 
program. 

1  2  3  4  5  6 

Strongly  Agree  Strongly  Disagree 


F:  4.  I  was  unfamiliar  with  much  of  the  terminology  used  by  the  diagnostic  program. 

1  2  3  4  5  6 

Strongly  Agree  Strongly  Disagree 


PU/T:  5.  Treatment  recommendations  suggest  I  do  things  that  I  can  not  do  at  sea. 

1  2  3  4  5  6 

Strongly  Agree  Strongly  Disagree 


PU:  6.  I  found  the  classification  of  diagnoses  into  probable  and  possible  helpful. 

1  2  3  4  5  6 

Strongly  Agree  Strongly  Disagree 


*The  responses  made  by  each  corspman  are  identified  by  the  letters  a-e.  Correct  interpretation 
requires  knowledge  of  the  individual’s  combined  ratings  for  these  two  questions. 
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PU:  11.  This  program  provides  a  good  supplement  to  the  dental  textbooks  kept  aboard 
submarines  for  the  management  of  dental  injuries  at  sea.* 

1  2  3  4  5  6 

Strongly  Agree  Strongly  Disagree 

*  Two  ratings  were  missing. 


PU:  12.  Aboard  submarines,  I  would  rather  have  all  dental  text  in  computer  rather  than  book 
format. 

1  2  3  4  5  6 

Strongly  Agree  Strongly  Disagree 
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SECTION  IV 


We  want  to  he**-your  comments  and  suggestions  regarding  the  computer  based  dental 
diagnostic  program.  Your  feedback  will  help  us  to  create  a  program  that  will  meet  the  needs 
of  the  corpsman. 

1.  Did  you  encounter  any  problems  in  using  this  program?  Please,  be  specific. 

No 

No 

A  couple  of  questions  requested  more  than  one  piece  of  information  and  could 
not  be  answered  with  a  "Yes"  or  "No"  response. 

I  could  not  back  up  to  previous  pages. 


2.  Do  you  have  any  suggestions  to  improve  the  computer  based  dental  diagnostic  program?  Be 
specific. 

Eliminate  soft  tissue  section. 

Leave  soft  tissue  section  out. 

Give  more  detail  to  definitions  of  terms  and  diseases. 

More  definitions. 


3.  What  do  you  consider  the  advantages  of  this  program? 

You  have  a  second  opinion  at  your  fingertips. 

The  program  keeps  me  flowing  in  the  right  direction  en  route  to  a  correct  diagnosis. 

I  liked  the  treatment  and  managment  recommendations. 

I  found  no  real  advantage.  If  the  patient  has  pain ,  then  give  pain  medication. 

If  the  patient  Mas  an  infection,  then  give  antibiotics.  The  corpsman’s  textbooks 
shows  how  to  treat  any  dental  problem. 

Excellent  back  up  system.  The  system  would  give  the  Commanding  and  Executive 
Officers  more  confidence  in  the  diagnosis. 
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4.  What  do  you  consider  the  disadvantages  of  this  program? 

The  corpsman-  will  rely  on  the  program  to  the  extent  he  is  unable  to  make  his  own 
diagnosis. 

Most  dental  emergencies  are  easily  seen  and  do  not  require  computer  assistance. 

I  don’t  think  it  justifies  the  expense.  All  of  my  patient’s  could  be  diagnosed  after 
one  glance  in  the  mouth. 
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